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Appendix I 

Semantic Tagging ("tagging") is a process of 
formatting individual questions and responses in a survey 
5 in a formal, machine-readable knowledge representation 
language (KRL) to enable automated analysis of data 
obtained via that survey. The semantic tags (or simply 
"tags") indicate the meaning of a response to a question in 
a particular way. The tags are created by a survey author 

10 (either a person, a computer program, or a combination 

thereof) and allow engine 30 to understand both a question 
and a response to that question. 

Tags indicate what the gathered information actually 
represents and allow the engine 30 to process data 

15 autonomously. In particular, the tags allow the data 

collected by a survey to be directly processed by an expert 
(i.e., rule-based) or logic programming (e.g., Prolog- 
based) system in engine 30 without requiring direct human 
intervention to interpret, categorize, summarize, etc., 

20 survey responses. User responses are asserted as facts 
within an expert system (e.g., within engine 30), where 
each fact is automatically derived from the tag associated 
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with each question. 

Tags represent the information gathered by a 
particular question, but are not tied to the precise 
wording of that question. Thus, it is possible for a wide 
5 range of natural language questions to have identical 
semantic tags. The KRL here has a partial ontology for 
describing survey questions and responses. It is intended 
to be descriptive and functional and thereby capture the 
vast majority of questions on marketing surveys. 

10 in this embodiment, surveys are comprised of three 

types of questions: behavioral, attitudinal, and 
demographic. Each of these question types has a 
corresponding unique type of tag which, as noted above, 
includes question and response fields. Examples of the 

is question fields of these tags are set forth below. In the 
tags, the following conventions apply: 



(1) 



I " represents a logical OR operation 



(2) 



plain text represents constants or list headers 



in an s-expression 



20 



(3) 



bold print represents keyword arguments 



(4) 



italics represent a member of a named set 



(5) 



<brackets> surround optional items 
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(6) $ { NAME } refers to a variable 



1 . 0 Behavioral Questions 

The question field tag template for behavioral 
5 questions is as follows. 



(tag (type behavioral) 

(time 

(tense past I present I future) 

io (startDate date) 

<(endDate date)> 

) 

(activity 

(action act I act AND act I act OR act) 
is (queryRegarding quality) 

(object product) 

< (subject demographic) > 

< (indirectObject demographic) > 

<(verb string)> 
20 < (variable string) > 

) 

<(questionID string) > 
(response ...) ) 



25 act e {Use, Do, Purchase, Replace, License, Own, Sell, 

Exchange, Recommend, Repair, Visit, Contact, Complain, and 
similar expressions } 



quality € {Frequency, Length, Existence, Source, 
30 Intention, Purpose, Completion, Difficulty, and similar 
expressions } 
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string is a "quotation delimited" string of characters. 

product is a set of products and/or services offered by a 
5 particular client and is industry specific. It is 

enumerated when the expert system is first installed for a 
client and subsequently can be modified to reflect the 
evolution of the client's product line or the client's 
industry as a whole. Elements of the product set have ad 
10 hoc internal structure representing both the client's 
identity and an item's position in the client's overall 
hierarchy of product/service offerings. By way of example, 
an IBM laptop computer is represented by 
" IBM/product/ computer / laptop . " 

15 

The demographic set is defined in the section of templates 
for demographic questions set forth below. 

The response field that corresponds to the above question 
20 field is specified below. 
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Each individual question in a survey has a tag that 
adheres to the above template, but need not assign optional 
fields. For example, consider the following behavioral 
questions and their associated tags immediately following 
5 the questions. 



(1) "Have you used an IBM laptop computer in the past 3 
years?" 



10 (tag (type behavioral) 

(time 

(tense past) 

(startDate ( $ { CURRENT_DATE } - 3 YEARS) ) 
( endDate $ { CURRENT_DATE } ) ) 
is (activity 

(action Use) 

(queryRegarding Existence) 

(object " IBM/product/ computer/ laptop") 

) 

20 (response 

(type YorN) ) ) 



(2) "How often do you replace your server?" 



25 (tag (type behavioral) 

(time 

(tense present) 

( startDate $ { CURRENT_DATE } ) 

) 

30 (activity 

(action Purchase) 
(queryRegarding Frequency) 
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(object "any/product/ server") 
) 

(response 

(type Selection) 

(selections (choseOne 0 3 6 9 12 18)) 
(primitivelnterval Month) ) ) 



(3) "What brand of laptop computer do you use now?" 



io (tag (type behavioral) 

(time 

(tense present) 

( startDate $ { CURRENT_DATE } ) 

) 

15 (activity 

(action Use) 
(queryRegarding Source) 

(object " current /product / computer / laptop") 
(variable "CURRENT J3RAND") 
20 ) 
(response 

(type MenuSelection) 

(selections (onlyOne "IBM" "Compaq" "NEC" "Gateway" 
"Dell" "Sony" "HP") ) 
25 (setVariable "CURRENT BRAND"))) 



2 . 0 Behavioral Questions 

The question field tag template for attitudinal 
30 questions is as follows. 



(tag (type attitudinal) 

(time 

(tense past I present I future) 

35 (startDate date) 
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<(endDate date)> 



(attitude 



10 



5 



(belief belief ) 
(queryRegarding belief Quality) 
< ( statement string) > 
< (subject demographic) > 
(object reference) 
(attribute feature) 
< (contrast reference) > 
< (variable string) > 



< (questionID string) > 
(response ...) ) 



15 



belief € {Satisfaction, Perception, Preference, Agreement, 
Modification, Plausibility, Reason, and similar 
expressions } 

20 belief Quality e {Degree, Correlation, Absolute, Ranking, 
Specification, Elaboration, and similar expressions} 

feature is a set of features relevant to a particular 
client's product and/or service offerings. Although many 
25 features are industry specific, many such as reliability 
are fairly universal. The feature set is enumerated when 
the expert system is first installed for a client and can 
be subsequently modified to reflect the evolution of the 
client's product line or industry as a whole. 
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The response field that corresponds to the above question 
field is specified below. 

5 It is noted that, for questions with matrix scales, 

which is a common occurrence in attitudinal questions, each 
row of the matrix has a separate, unique tag. 

Consider the following attitudinal questions and their 
10 associated tags, which immediately follow the questions. 



(1) "Rate your overall satisfaction with the performance 
of the. laptop computer you are currently using." 



is (tag (type attitudinal) 

(time 

(tense present) 

(start Date $ { CURRENT_DATE } ) 

) 

20 (attitude 

(belief Satisfaction ) 
(queryRegarding Degree) 

(object "cur rent /product /laptop/ computer") 

(attribute Performance) ) 
25 (response 

(type Horizon talLikert) 

(askingAbout Satisfaction) 

(selections (low 0) 
(high 5) 
30 (interval 1) ) ) 
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(2) "If you could make one change to your current laptop 
computer, what would it be?" 



(tag (type attitudinal) 

5 (time 

(tense present) 

(start Date $ { CURRENT_DATE } ) 

) 

(attitude 

io (belief Modification ) 

(queryRegarding Specification) 

(object "current/product/ laptop/ computer") ) 
(response 

(type ListSelection) 
15 (selections (onlyOne ${ FEATURES })) ) 



(3) "Do you agree with the sentiment that laptop computers 
will someday replace desktop computers?" 



20 (tag (type attitudinal) 

(time 

(tense present) 

(start Date $ { CURRENT_DATE } ) 

) 

25 (attitude 

(belief Agreement ) 

(queryRegarding Absolute) 

(obj ect " any /product / laptop/ computer ") 

(statement "Laptop computers will someday replace 

30 desktop computers.")) 

(response 

(type YorNorDontKnow) 
(askingAbout Agreement) ) 



35 (4) "Do you have any additional comments to add?" 
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(tag (type attitudinal) 

(time 

(tense present) 
5 ( startDat e $ { CURRENT_DATE } ) 

) 

(attitude 

(belief Perception ) 
(queryRegarding Elaboration) 

10 ( ob j ect "any/product/ laptop/ computer") 

) 

(response 

( type FreeResponse) 

(noLines 3) 
15 (width 40) 

) 

) 



20 3.0 Demographic Questions 

The question field tag template for demographic 
questions is as follows: 



(tag (type demographic) 

25 (time 

(tense past I present I future) 

(startDate date) 
<(endDate date)> 

) 

30 (description 

< (gender) > 

< (age) > 

< (ageRange) > 

< (haveChildren) > 
35 < (numberChildren) > 

< (childAgeByRange) > 

< (maritalStatus) > 
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< 


(employment) > 




< 


[education) > 




< 


(income) > 




< 


[address ) > 


5 


< 


[ ema i 1 ) > 




< 


( name ) > 




< 


(phoneNumber ) > 




< 


(faxNumber) > 




< 


(city) > 


10 


< 


(state) > 




< 


( zipCode) > 




< 


(publicationsRead) > 




< 


(groupMembership) > 




< 


(hobbies) > 


15 


< 


(mediaOutlets) > 




< 


(other string)> 



< (qualifier length I prefer I like I dislike 
I know I dontKnow) > 
) 

20 <(questionID number) > 

(response ...) ) 



The response field for the above question field is 
specified below. 



By way of example, consider following the demographic 
questions and their associated tags immediately following 
the questions. 



30 (1) "What is your gender?" 



(tag (type demographic) 

(time 

(tense present) 
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( startDate $ { CURRENT_DATE } ) ) 
(description 

(gender) ) 
(response 

(type Selection) 

(selections (onlyOne "Male" "Female")))) 



(2) "What is your email address?" 



10 (tag (type demographic) 

(time 

(tense present) 

( startDate $ { CURRENT_DATE } ) ) 
(description 
15 (email) ) 

(response 

(type FreeResponse) 
(noLines 1) 
(width 30) 

20 ) 



(3) "How long have you lived at your present address?" 



25 (tag (type demographic) 

(time 

(tense present) 

(startDate $ { CURRENT_DATE } ) ) - 
(description 
30 (address) 

(qualifier length) 

) 

(response 

(type MenuSelection) 

35 (low 0) 

(high 20+) 
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(primitive Interval Year) 



5 



4 . 0 Response Field Template 

Questions in surveys can have a variety of different 
scales for allowing the respondent (i.e., the one taking 
the survey) to select an answer. The response field of a 

io tag specifies, for each question in the survey, both the 
general scale-type that the response field uses and how to 
instantiate that scale to obtain a valid range of answers. 

The response field also contains placeholders for the 
respondent's actual answers and individual (perhaps 

is anonymous) identifier ( s) . Each completed survey for some 
respondent leads to all of the tags associated with that 
survey being asserted as facts in the expert system, with 



all of the placeholders appropriately filled in by the 
respondents 1 answers. For expert systems, such as CLIPS 
20 (C-Language Integrated Product System) that do not support 
nested structures within facts, the actual data 
representation is a flattened version of the one shown 
below. 
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A representative template for the response field is as 
follows . 



(response 
5 (type scale) 

< (askingAbout questionTopic) > 

< (prompt string) > 

< (low number) > 
<(high number) > 

10 < (interval number)> 

< (scaleLength number) > 

< (primitivelnterval time I distance I temperature) > 
< (selections (onlyOne | anyOf string*) 
<(upto number) > 
15 <(atLeast number)> 

)> 

< (width number) > 

< (noLines number) > 

20 (userSelectionRaw string I number) 

(user Select ion string) 

(user Select ionType string) 

(userlD number) 

(userlDinternal number) 
25 (userlDconf idential string) 

(clientID string) ) 



scale € {Likert, Selection, MenuSelection, YorN, 
YorNorDontKnow, FreeResponse, HorizontalLikert } 



The askingAbout field can be set to have the expert system 
automatically generate the prompt for selecting an answer. 



questionTopic e {Preference, Sentiment, Belief, Frequency, 
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Comment , and similar expressions} 

5 . 0 Fact Instantiation from Tags 
5 Tags allow the data collected by a survey to be 

directly processed by an expert (i.e., rule-based) or logic 
programming (e.g., Prolog-based) system (engine 30) without 
requiring direct human intervention to interpret, 
categorize, summarize, etc., survey responses. User 

10 responses are asserted as facts within the expert system, 
where each fact is automatically derived by parsing the 
relevant information from a corresponding tag associated 
with each question. 

It is noted that that additional information regarding 

is each user is simultaneously instantiated in separate facts 
within the expert system. This includes for example, the 
site where the respondent was surveyed, the time of day the 
survey was taken, and the like. 

By way of example, consider the question: 

20 

"How often do you speak with your salesman?", 
with associated tag: 
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(tag 



(type behavioral) 



(time 



(tense current) 

(startDate $ { CURRENT DATE}) 



10 



15 



25 



(activity 

(action Contact) 

(queryRegarding Frequency) 

(indirectObj ect "NEC/person/ salesman") 

(object "NEC/product/ 'PBX/NEAX2000") ) 
(response 

(type Selection) 

(askingAbout Frequency) 

(selections (onlyOne 0 3 6 9 12 18)) 

(primitivelnterval Month) ) ) 



If a respondent answering this question selects "3", 
as in, "I speak with my salesman every 3 months", the 
expert system will automatically assert a fact 
corresponding to the tag in the expert system, with 
additional fields representing the user's selection and 
identity, as well as identifying information about the 
survey itself. This is set forth as follows. 



(surveyName "PBX Satisfaction") 

(surveyDate 12/17/00) 

(survey Version "1 . 0") 

(questionID 3) 

(type behavioral) 

(time 

(tense current) 

(startDate 12/17/00) 



(answer 
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) 

(activity 

(action Contact) 

(queryRegarding Frequency) 
5 (indirectObject "NEC/person/salesman") 

(object "NEC/product/PBX/NEAX2000")) 
(response 

(type Selection) 

(askingAbout Frequency) 
10 (selections (choseOne 0 3 6 9 12 18)) 

(primitivelnterval Month) 

(userSelectionRaw 3) 
(userSelection 3) 
15 (userSelectionType Month) 

(userlD 127) 
(userlDinternal 4208) 

(userlDconf idential "mhcoen@intellistrategies . com : uid 
0xcf023a8b7") 
20 (client "NEC/CNG") 

) 

In this way, engine 30 is able to interpret the 
responses to survey questions using tags. The response 
25 information is analyzed, as described above, to generate 
graphical displays and recommend follow-up surveys. 
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Appendix II 

Over-performance and under-perf ormance graphs are 
components of the report card. The under-perf ormance 
5 display is generated according to the process in section 
1.0 below and the over-performance display is generated 
according to the process in section 2.0 below. 



1 . 0 Under- Performance Display 

io For client company b (who is running engine 30) & 
For each competitor company c & 
For each feature f & 
For each user u 

15 Such that we know: 



(1) (importance of f to u) 

(2) (satisfaction rating of company b on feature f to 

user u) 

20 (3) (satisfaction rating of company c on feature f to 

user u) 

(4) (all involved data is less than 2 months old) 



Calculate : 

25 (1) average and variance of satisfaction for each 

feature over all competitors c) 
Call these quantities avg(f) and stddev (f) 
respectively 

(2) (average of satisfaction for each feature for 
3 0 company b) 

Call this quantity avg(f,b) 



- 68 - 



Attorney Docket No. 11464/002001 

Sort features by importance and proceed through them in 
decreasing order: 

If (avg(f) - avg(f,b) > stddev (f)) 
5 Then set rank(f) = (sqrt (importance (f) )* (avg (f) - 

avg (f ,c) ) ) - penalty (avg (f) , stddev (f) *2) 

We also subtract a penalty term from the rank(f) to 
discount features with high variance either at the moment 
10 (as shown here) or historically. 

Loop: Consider the n features with the highest rank, where 
n is the number of features to be displayed in the under- 
performance graph. If any of them are proxies for a 
is derived attribute, here a feature, and the other proxy 
attributes are known, calculate the rank for the derived 
feature and use it instead. 

Go to Loop. 

20 if not, continue. 

Generate a chart or graph for each feature and display the 
features in reverse order by rank. 
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2 . 0 Over- Performance Display 

For client company b (who is running engine 30) & 

For each competitor company c & 

For each feature f & 

5 For each user u 



Such that we know: 

(1) (importance of f to u) 

(2) (satisfaction rating of company b on feature f to 
io user u) 

(3) (satisfaction rating of company c on feature f to 

user u) 

(4) (all involved data is less than 2 months old) 



15 Calculate: 



(1) (average and variance of satisfaction for each 

feature over all competitors c) 
Call these quantities avg(f) and stddev (f) 
respectively 

20 (2) (average of satisfaction for each feature for 

company b) 
Call this quantity avg(f,b) 

Sort features by importance and proceed through them in 
25 increasing order: 



If (avg(f,b) - avg(f) > stddev (f)) 

Then set rank(f) = (sqrt(max - importance (f) )* (avg (f) 
- avg(f,c))) - penalty (avg (f) , stddev(f) *2) 



We also subtract a penalty term from the rank to discount 
features with high variance either at the moment (as shown 
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here) or historically. Max represents the maximum feature 
value (i.e., as determined by the source question's scale). 

Loop: Consider the n features with the highest rank. (n is 
5 the number of features to be displayed in the 

underperf ormance graph) . If any of the n features are 
proxies for a derived attribute (here a feature) and the 
other proxy attributes are known, calculate the rank for 
the derived feature and use it instead. 

10 

Go to Loop. 

If not, continue. 

Generate chart for each feature and display them in reverse 
is order by rank. 
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Appendix III 

Surveys by nature are very specific documents. They 
are written with respect to a particular inquiry, to a 
5 specific industry (or entity), to a particular product, 
offering, or concept, and for an intended audience of 
respondents. These determine not only the structure of the 
overall survey but the particular choice of wording in the 
questions and the structure, wording, and scale of the 

10 question answers. 

The system (engine 30) has a library of surveys that 
it can deploy, but instead of containing the actual text of 
each of their questions, the surveys contain question 
templates. Each of these templates captures the general 

is language of the question it represents without making any 
commitment to certain particulars. The system fills in the 
details to generate an actual question from a template 
using an internal model of the client who is running the 
survey that is created during engine 30 ! s configuration for 

20 that client. This model includes the client's industry, 
product lines, pricing, competitors, unique features and 
offerings, resellers, demographic targets, customer 
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segmentations, marketing channels, sales forces, sales 
regions, corporate hierarchy, and retail locations, as well 
as general industry information, such as expected time 
frames for product/service use, consumption, and 
5 replacement. 

Although generating the question templates requires 
more effort than simply writing questions directly, it 
avoids the effort of customizing and modifying every The 
system survey for each new client. 
10 The following are examples of survey questions and the 

templates that generate them: 



1) Purchase frequency: 

a. How many laptop computers have you purchased in the 
is past 10 years? 

b. How many airlines tickets do you buy per year? 



(question (variables $ { CURRENT_PRODUCT } 
$ { PURCHASE_INTERVAL } ) 
20 (text 

"How many $ { CURRENT_PRODUCT } " 
(if $ { PURCHASE_INTERVAL == 12) {"do you buy per 

year" } 

elseif ((mod $ { PURCHASE_INTERAL } 12) == 0) 

25 {"have you bought in the. past " 

( $ { PURCHASE_INTERVAL } / 12) 

" years"} 

else {"have you bought in the past 

$ { PURCHASE_INTERVAL } months"} 

30 ) 

« 9 // 
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2) Competitive Position/reliability: 

a. Which brand of PBX do you think is most reliable? 
5 □ NEC 

□ Nortel 

□ Lucent 

□ Williams 

io jb. Which type of vehicle do you think is most 

reliable? 

□ Pickup Truck 

□ SUV 

□ Station wagon 
15 □ Sedan 



(question (variables $ { CATEGORY_REFERENCE } 
$ { CURRENT_PRODUCT } 
2 0 ${ MANUFACTURERS } ) 

(text 

"Which $ { CATEGORY__REFERENCE } of 
${CURRENT_PRODUCT} do you think is most reliable? 
) 

25 (scale (selections $ {MANUFACTURERS } ) 

) 

) 



30 
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